1. Field
The present description relates to a method, system, and computer program product for error routing in a multi-root communication fabric.
2. Description of Related Art
There are various communication protocols for interconnecting peripheral devices such as input/output (I/O) adapters to a computing device such as a server. For example, the PCI Express (PCIe) is a protocol which is widely used in computer systems to interconnect servers to adapters or other components, often by means of a PCIe switch or a cascade of PCEe switches.
A storage controller may have multiple servers often referred to as a cluster of servers, which receive input/output requests from one or more hosts, to perform input/output operations in which data is read from or written to storage through various I/O adapters. Each cluster may have one or more central processing units (CPUs) in which the processing and memory resources of the cluster may be apportioned into logical partitions, each of which is capable of running an operating system and performing the functions of a “server”. Thus, as used herein, the term “server” may be used to refer to a physical server or a logical partition performing a server function.
A server may have multiple I/O adapters including host and device adapters which are accessed through a switch such as a PCIe switch. In one known design, the switch has one “upstream port” connected to the server, and several downstream ports connected to the various I/O adapters. These are referred to as transparent ports.
To increase efficiency, it is often desirable to share I/O adapters amongst the servers of the cluster. Thus, a device adapter, for example, may be shared as a “virtual” device adapter. The servers typically communicate with the device adapters and other I/O adapters over a “fabric” which may comprise one or more interfaces providing communication paths between the servers and adapters.
However, each I/O adapter typically is assigned a particular server which is designated the “root server” or “root node” for that I/O adapter. A group of I/O adapters may share a root server. One function of the root server is to control the associated I/O adapters including handling error messages from I/O adapters controlled by the root server. Error messages, in the form of packets in the PCIe protocol, provide notice of errors occurring in the shared I/O adapter or in the fabric, and are typically handled with special requirements. More particularly, such error messages often need to be routed to the root server to which the I/O adapter is assigned and by which the adapter is controlled.
The fabric may include one or more switches which permit the I/O adapters to be shared by the servers. Each switch typically includes routing logic which logically couples a selected upstream port of the switch to a selected downstream port of the switch. In this manner, a particular server coupled to an upstream port, may be logically coupled to a particular device adapter coupled to a downstream port, for the purpose of conducting I/O operations between that server and storage via the device adapter. Similarly, a particular server coupled to an upstream port, may be logically coupled to a particular host coupled to a downstream port, for the purpose of conducting I/O operations between that server and host via the host adapter. Thus, each server may be logically coupled to each I/O adapter of the shared adapters via the switch.
Each switch typically further includes error reporting logic. In the course of an I/O operation, an error may be detected by the switch or one or more of the shared I/O or host adapters. The switch is configured or programmed to report the occurrence of an error via the “transparent” upstream port which is the upstream port connected to the root server for the switch and the shared I/O adapters coupled to the switch.
It is believed that the upstream port connected to the root server is typically configured as a transparent port by the root server when the switch and the I/O adapters connected to the switch are initialized. Thus, the switch and the shared I/O adapters connected to the switch may be connected to the fabric and powered on to initial hardware and software startup states following power on. The server designated to be the root server for the switch and its associated I/O adapters detects the presence of the switch and the I/O adapters in the fabric and initializes the switch and the I/O adapters using appropriate driver software in the server.
Thus, at startup, the driver software of the root server stores configuration data in the switch to configure a port as the transparent upstream port if it was not already so configured as a transparent upstream port by default. In addition, the root server configures additional ports as “non-transparent” ports if not already configured as non-transparent upstream ports by default. In some PCIe switch designs, a downstream port may be connected to a server and configured as non-transparent port. The server connected to the non-transparent port sees that port as an endpoint. In this manner, the non-transparent port connected to a server logically functions as a non-transparent upstream port.
Hence, each of the servers may access each of the initialized I/O adapters via the initialized switch to conduct I/O operations. Conversely, each of the I/O and host adapters may access each of the servers to conduct I/O operations such as direct memory access (DMA) operations. However, the I/O adapters are controlled by the root server which initialized the switch and the I/O adapters at startup.
In the event that the root server fails, the I/O adapter hardware and software may be reset. During such a reset, the I/O adapter hardware and software may be returned to an initial startup state. The reset I/O adapters may be reinitialized by another server so that the reinitialized I/O adapters are controlled by a different server which becomes a new or successor root server for those I/O and host adapters. In connection with the reinitialization, the switch may be reinitialized so that the switch port connected to the successor root server and functioning as an upstream port is configured to be the transparent upstream port.
Various types of errors may occur in an I/O adapter such as a device adapter. For example, many communication systems employ a periodic cyclic redundancy check (CRC) to detect alteration of data being communicated. Thus, I/O transactions to a device adapter may have CRC errors, may be malformed, poisoned, timed out, aborted, unexpected, unsupported, etc. In a PCI Express fabric, in response to a detected error, details of the PCI transaction in which the error arose may be logged in one or more PCI standard advanced error reporting (AER) registers. In some applications, a generic error message may be sent to the root server by the switch via the transparent upstream port.